Evaluating QA Systems on Multiple Dimensions
نویسندگان
چکیده
Question-answering systems are expanding beyond information retrieval and information extraction, to become fullfledged, complex NLP applications. In this paper we discuss the evaluation of question-answering systems as complex NLP systems, and suggest three different dimensions for evaluation: objective or information-based evaluation; subjective evaluation; and architectural evaluation. We also discuss the role of ambiguity resolution in QA systems, and how ambiguity resolution might be evaluated.
منابع مشابه
A New Statistical Model for Evaluation Interactive Question Answering Systems Using Regression
The development of computer systems and extensive use of information technology in the everyday life of people have just made it more and more important for them to make quick access to information that has received great importance. Increasing the volume of information makes it difficult to manage or control. Thus, some instruments need to be provided to use this information. The QA system is ...
متن کاملNew Measures for Open-Domain Question Answering Evaluation Within a Time Constraint
Previous works on evaluating the performance of Question Answering (QA) systems are focused on the evaluation of the precision. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA systems within a time constraint in the CLEF-2006 campaign, using the pro...
متن کاملEvaluación de Sistemas de Búsqueda de Respuestas con restricción de tiempo
Previous works on evaluating the performance of Question Answering (QA) systems are focused in the evaluation of the precision. Nevertheless, the importance of the answer time never has been evaluated. In this paper, we developed a mathematic procedure in order to explore new evaluation measures in QA systems considering the answer time. Also, we carried out an exercise for the evaluation of QA...
متن کاملEvaluating Answer Validation in Multi-stream Question Answering
We follow the opinion that Question Answering (QA) performance can be improved by combining different systems. Thus, we planned an evaluation oriented to promote the specialization and further collaboration between QA systems. This multistream QA requires to develop the modules able to select the proper stream according to the question and the candidate answers provided. We describe here the ev...
متن کاملQualitative Dimensions in Question Answering: Extending the Definitional QA Task
Current question answering tasks handle definitional questions by seeking answers which are factual in nature. While factual answers are a very important component in defining entities, a wealth of qualitative data is often ignored. In this incipient work, we define qualitative dimensions (credibility, sentiment, contradictions etc.) for evaluating answers to definitional questions and we explo...
متن کامل